fix(artifacts): make stored DataFrames servable a page at a time by tkislan · Pull Request #142 · deepnote/deepnote-toolkit

tkislan · 2026-10-24T08:14:19Z

Summary

Fixes #141.

store_dataframe now writes Arrow files that the dataframe reader can serve a page at a time. The toolkit cuts the record batches itself, records each batch's row count in the manifest as batch_rows, caps batches by bytes as well as rows, and compresses with zstd or not at all.

Changes

Writing (`_dataframe_storage.py`)

pandas and polars share one write path. The frame becomes an Arrow table (pa.Table.from_pandas(df, preserve_index=None) for pandas, df.to_arrow(compat_level=oldest) for polars), and pa.ipc.new_file writes it one batch at a time.
Rows per batch is clamp(8 MiB / average row bytes, 1, 65,536).
- Row bytes are summed per column with nbytes, because every batch writes its own copy of each column and chunk, even ones that share buffers.
- A column holding a dictionary uses min(nbytes, get_total_buffer_size()), because the file holds the dictionary once.
A batch over 12 MiB, which long rows that sit together can produce, is split into the longest prefixes that fit. A row over 12 MiB gets a batch of its own.
Each batch is a slice of the table with its chunks combined, so small chunks don't make small batches.
Dictionaries are unified before writing, because an IPC file can't replace a column's dictionary.
Compression is zstd when pyarrow has it, else none. Never lz4.

polars checks (`_polars_compatibility_checks.py`, new)

Each raises TypeError before anything is written:

polars older than 1.35.1: its to_arrow() copies a Categorical's whole dictionary into every chunk.
Object columns, including Object nested in a List, Array or Struct: to_arrow() exports each value as its memory address.
Array columns at any depth, on polars older than 1.43.2: to_arrow() applies a sliced Array's offset twice when the slice holds a null (Exporting a sliced Array column with nulls over the Arrow C interface returns shifted values; pa.table(df) aborts the process pola-rs/polars#28583).

Manifest (`_dataframe_storage_manifest.py`)

New required field batch_rows; rows is its sum.
The size cap rises from 4 KiB to 64 KiB. encode_manifest raises ValueError past the cap, before the current frame is replaced. That takes about 9,000 batches, some 600M rows.

`dataframes.py`

The store_dataframe docstring lists the new errors.

Behaviour changes

A manifest without batch_rows, as written by 2.9.0, counts as absent. The next store of that name doesn't keep its file as previous, and the file is deleted once it is a day old, like any unreferenced frame.
Batch boundaries change. Values, schema and pandas metadata don't.
New errors: the polars checks above, and ValueError for a frame that needs more batches than the manifest can list. polars Int128 columns raise ArrowInvalid from to_arrow().
polars frames are converted whole by to_arrow(), which also copies string views to large_string. Storing a 1.4 GiB frame of 10M strings now raises peak memory by 1.47 GiB, up from 0.35 GiB. pandas is unchanged.
A polars frame of about 100k chunks stores about 20 times slower, because to_arrow() costs about 0.09 ms per chunk.

Known limits

Categoricals stay dictionary-encoded, and the reader reads every dictionary for every page. Dictionaries past the reader's limit of about 1M values make every page unservable.
On polars older than 1.43.0, a sliced Struct column holding a null makes store_dataframe raise pyarrow's ArrowInvalid (a polars bug). The previous frame is kept.
The dataframe reader must accept manifests up to 64 KiB.

Testing

Unit tests in tests/unit/artifacts/ cover:
- batch_rows matching the file's record batches;
- the 8 MiB byte cap, and a row larger than a batch;
- long rows that sit together, split into batches of at most 12 MiB without a trail of small ones;
- buffers shared across columns and chunks;
- dictionary unification;
- zstd-or-none compression;
- the manifest size cap and batch_rows validation;
- each polars check, and frames the checks allow.
End to end, the dataframe reader (with its manifest cap raised to 64 KiB) served every page at page sizes 100 and 1000 for frames written by this branch:
- ints and categoricals;
- chunked polars frames;
- columns and chunks that share buffers;
- empty frames and frames without columns;
- long text;
- mixed types.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VNWoySEbnsryRwTGKUydNx

Summary by CodeRabbit

New Features
- Dataframe storage now writes data in sized batches and tracks each batch’s row count. Zstandard compression is used when available; otherwise data is stored without compression.
- Manifest capacity has increased to 64 KiB, supporting larger stored dataframes.
Bug Fixes
- Clearer errors are provided when Polars versions or dataframe column types are unsupported. Existing stored data is preserved if an incompatible dataframe cannot be saved.
- Read-only storage mounts continue to avoid converting dataframes when saving is unavailable.
Documentation
- Documented storage limits and the Polars version and data type requirements.

… DataFrames store_dataframe wrote frames that could be loaded whole but not served a page at a time. The manifest had no per-batch row counts, batches were capped by rows only, and compression could fall back to lz4. - Write pandas and polars frames through one pyarrow IPC path that cuts the record batches itself and records their row counts in the manifest as batch_rows. - Cap batches at about 8 MiB as well as 65,536 rows, sizing them from the average row, and merge small chunks into whole batches. - Compress with zstd, else not at all. - Raise the manifest cap to 64 KiB, and refuse a frame whose manifest would exceed it. - Refuse polars Object columns, nested ones included. Rechunk chunked Categoricals on polars before 1.35.1. Fixes #141 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VNWoySEbnsryRwTGKUydNx

github-actions · 2026-10-24T08:14:53Z

📦 Python package built successfully!

Version: 2.9.0.dev17+584b026
Wheel: deepnote_toolkit-2.9.0.dev17+584b026-py3-none-any.whl

Install:

pip install "deepnote-toolkit @ https://deepnote-staging-runtime-artifactory.s3.amazonaws.com/deepnote-toolkit-packages/2.9.0.dev17%2B584b026/deepnote_toolkit-2.9.0.dev17%2B584b026-py3-none-any.whl"

codecov · 2026-10-24T08:18:11Z

Codecov Report

❌ Patch coverage is 98.07692% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.39%. Comparing base (6d93e12) to head (bac0397).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
deepnote_toolkit/artifacts/_dataframe_storage.py	96.77%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #142      +/-   ##
==========================================
+ Coverage   78.17%   78.39%   +0.21%     
==========================================
  Files         120      121       +1     
  Lines        6837     6906      +69     
  Branches      990     1002      +12     
==========================================
+ Hits         5345     5414      +69     
  Misses       1182     1182              
  Partials      310      310

Flag	Coverage Δ
combined	`78.39% <98.07%> (+0.21%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Before 1.43.2, polars' to_arrow() applies the offset of a sliced Array column with nulls twice (pola-rs/polars#28583), so store_dataframe stored shifted values, or a file that aborts pyarrow on read. On those versions, a frame with an Array column at any depth now raises TypeError with an upgrade hint. Struct had the same bug before 1.43.0, but pyarrow rejects that export, so storing fails instead of writing wrong values; Struct frames are not refused. polars older than 1.35.1 is refused outright. That removes the rechunk of chunked Categoricals before 1.35.1, the plain to_arrow() branch before 1.2.1, and the polars.Enum lookup that raised AttributeError on polars older than 0.20. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VNWoySEbnsryRwTGKUydNx

deepnote-bot · 2026-10-26T19:50:55Z

🚀 Review App Deployment Started

📝 Description	🌐 Link / Info
🌍 Review application	ra-142
🔑 Sign-in URL	Click to sign-in
📊 Application logs	View logs
🔄 Actions	Click to redeploy
🚀 ArgoCD deployment	View deployment
⏰ Last deployed	2026-10-29 18:37:28 (UTC)
📜 Deployed commit	`a9b04e2b4690493a3723e92e8bf22c9229de459b`
🛠️ Toolkit version	`584b026`

Split the polars version, Object and Array checks out of _to_arrow_table, and reword the lz4 note in _select_compression. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VNWoySEbnsryRwTGKUydNx

…r own modules Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VNWoySEbnsryRwTGKUydNx

…ions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VNWoySEbnsryRwTGKUydNx

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VNWoySEbnsryRwTGKUydNx

new_file writes nothing; the first bytes reach the file in write_batch. The failing mocks now let the real write_batch write a batch, then raise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VNWoySEbnsryRwTGKUydNx

…taframes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VNWoySEbnsryRwTGKUydNx

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VNWoySEbnsryRwTGKUydNx

…d polars DataFrames

Batches are cut by the frame's average row size, so long rows that sit together could make one batch several times the 8 MiB target, and the dataframe reader refuses any page that reads more than 32 MiB. A batch over 12 MiB is now split into the longest prefixes that fit. Halving it instead would leave a trail of ever-smaller batches around the long rows, and the reader refuses a page that spans more than 8. Also wraps a test annotation that black rejected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VNWoySEbnsryRwTGKUydNx

tkislan · 2026-10-29T18:31:37Z

@coderabbitai full review

coderabbitai · 2026-10-29T18:31:44Z

✅ Action performed

Full review finished.

coderabbitai · 2026-10-29T18:31:52Z

No actionable comments were generated in the recent review.

Recent review info

Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: CHILL
Plan: Essentials
Run ID: c400ff8e-0075-4f80-923b-7f2ce35df4d3

Commits

Reviewing files that changed from the base of the PR and between 6d93e12 and bac0397.

Files selected for processing: 13

deepnote_toolkit/artifacts/_dataframe_storage.py
deepnote_toolkit/artifacts/_dataframe_storage_manifest.py
deepnote_toolkit/artifacts/_polars_compatibility_checks.py
deepnote_toolkit/artifacts/dataframes.py
tests/unit/artifacts/conftest.py
tests/unit/artifacts/test_arguments.py
tests/unit/artifacts/test_file_format.py
tests/unit/artifacts/test_manifest_and_cleanup.py
tests/unit/artifacts/test_polars_compatibility_checks.py
tests/unit/artifacts/test_read_only_mount.py
tests/unit/artifacts/test_storing.py
tests/unit/artifacts/test_write_failures.py
tests/unit/helpers/dataframe_storage.py

Included review availability: This review used your included allowance. 2 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 3 reviews per hour.

Walkthrough

The PR changes dataframe storage to use a shared Arrow IPC path for pandas and Polars frames. It adds dictionary unification, byte- and row-based batch sizing, oversized-row handling, and zstd-only compression selection. Manifests now store per-batch row counts and support up to 64 KiB. Polars compatibility checks reject unsupported versions and nested dtypes. Tests cover batching, manifests, cleanup, failures, and read-only mounts.

Priority: Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant store_dataframe
  participant Arrow IPC writer
  participant Manifest
  Client->>store_dataframe: store pandas or Polars frame
  store_dataframe->>Arrow IPC writer: convert and write sized record batches
  Arrow IPC writer-->>store_dataframe: return batch row counts
  store_dataframe->>Manifest: encode rows and batch_rows
  Manifest-->>store_dataframe: return encoded manifest
  store_dataframe-->>Client: complete storage or return None on EROFS

Possibly related issues

deepnote/deepnote-toolkit#141 — The changes add per-batch row counts, byte limits, compatible compression, and a larger manifest limit.

Suggested reviewers: m1so, mfranczel

Merge Risk: Minimal · up to bac03

No established defect blocks merging. Confirm that the deployed reader accepts 64 KiB manifests as part of normal integration checks.

Pre-merge checks: 4 · 1 · 1

Failed checks: 1 warning · 1 inconclusive

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 70.83% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…	Write docstrings for the functions missing them to satisfy the coverage threshold.
Updates Docs		The PR updates the `store_dataframe` API docstring in `deepnote_toolkit/artifacts/dataframes.py`. No repository documentation page references this feature. The required `deepnote/deepnote` OSS documen…	Update or verify the feature documentation in the `deepnote/deepnote` OSS repository and the roadmap in the `deepnote/deepnote-internal` landing page. Re-run the documentation check with access to those repositories.

Passed checks: 4 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main change: enabling stored DataFrames to be served one page at a time.
Linked Issues check		Issue `#141` coding requirements are implemented. `_dataframe_storage.py` converts pandas and polars frames to Arrow, writes toolkit-controlled batches, merges chunks, targets 8 MiB, caps batches at 65,…
Out of Scope Changes check		The changes stay within issue `#141`. The polars compatibility checks, public error documentation, manifest validation, cleanup handling, and tests directly support reliable page-readable storage and th…

Full details: Linked Issues check

Explanation

Issue #141 coding requirements are implemented. _dataframe_storage.py converts pandas and polars frames to Arrow, writes toolkit-controlled batches, merges chunks, targets 8 MiB, caps batches at 65,536 rows, splits oversized batches, and selects zstd or no compression. _dataframe_storage_manifest.py records batch_rows, derives rows, and enforces a 64 KiB limit before manifest replacement. Missing batch_rows makes older manifests unusable, so 2.9.0 files are not migrated until a new store.

Full details: Out of Scope Changes check

Explanation

The changes stay within issue #141. The polars compatibility checks, public error documentation, manifest validation, cleanup handling, and tests directly support reliable page-readable storage and the required failure behavior. No unrelated product behavior appears in the reviewed changes.

Full details: Docstring Coverage

Explanation

Docstring coverage is 70.83% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 24 functions across 13 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Full details: Updates Docs

Explanation

The PR updates the store_dataframe API docstring in deepnote_toolkit/artifacts/dataframes.py. No repository documentation page references this feature. The required deepnote/deepnote OSS documentation and private landing-page roadmap are not available in this checkout, so their update status cannot be verified.

Resolution

Update or verify the feature documentation in the deepnote/deepnote OSS repository and the roadmap in the deepnote/deepnote-internal landing page. Re-run the documentation check with access to those repositories.

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

tkislan and others added 10 commits October 27, 2026 08:59

Merge branch 'main' into tk/servable-stored-dataframes

e917870

refactor(artifacts): move the polars checks and their tests into thei…

2480edc

…r own modules Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VNWoySEbnsryRwTGKUydNx

refactor(artifacts): describe the manifest fields with Field descript…

c7a5640

…ions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VNWoySEbnsryRwTGKUydNx

test(artifacts): type the patched writers' sink as BufferedWriter

a76d0a1

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VNWoySEbnsryRwTGKUydNx

Merge remote-tracking branch 'origin/main' into tk/servable-stored-da…

692907a

…taframes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VNWoySEbnsryRwTGKUydNx

test(artifacts): replace Any with concrete types in the artifact tests

7dbd1f2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VNWoySEbnsryRwTGKUydNx

fix(tests): update type hint for make_frame to support both pandas an…

e7ac53c

…d polars DataFrames

coderabbitai Bot approved these changes Oct 29, 2026

View reviewed changes

tkislan marked this pull request as ready for review October 29, 2026 18:39

tkislan requested a review from a team as a code owner October 29, 2026 18:39

tkislan requested a review from m1so October 29, 2026 18:39

Navigation Menu

Repository navigation

fix(artifacts): make stored DataFrames servable a page at a time - #142

fix(artifacts): make stored DataFrames servable a page at a time#142
tkislan wants to merge 12 commits into
mainfrom
tk/servable-stored-dataframes

tkislan commented Oct 24, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

github-actions Bot commented Oct 24, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Oct 24, 2026 •

edited

Loading

Uh oh!

deepnote-bot commented Oct 26, 2026 •

edited

Loading

Uh oh!

tkislan commented Oct 29, 2026

Uh oh!

coderabbitai Bot commented Oct 29, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Oct 29, 2026 •

edited

Loading

Sequence Diagram(s)

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

tkislan commented Oct 24, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Writing (_dataframe_storage.py)

polars checks (_polars_compatibility_checks.py, new)

Manifest (_dataframe_storage_manifest.py)

dataframes.py

Behaviour changes

Known limits

Testing

Summary by CodeRabbit

Uh oh!

github-actions Bot commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

deepnote-bot commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tkislan commented Oct 29, 2026

Uh oh!

coderabbitai Bot commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Sequence Diagram(s)

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tkislan commented Oct 24, 2026 •

edited by coderabbitai Bot

Loading

Writing (`_dataframe_storage.py`)

polars checks (`_polars_compatibility_checks.py`, new)

Manifest (`_dataframe_storage_manifest.py`)

`dataframes.py`

github-actions Bot commented Oct 24, 2026 •

edited

Loading

codecov Bot commented Oct 24, 2026 •

edited

Loading

deepnote-bot commented Oct 26, 2026 •

edited

Loading

coderabbitai Bot commented Oct 29, 2026 •

edited

Loading

coderabbitai Bot commented Oct 29, 2026 •

edited

Loading